Is There a Problem With Evidence in Health Professions Education?

ABSTRACT: What constitutes evidence, what value evidence has, and how the needs of knowledge producers and those who consume the knowledge produced as evidence might be better aligned are questions that continue to challenge the health sciences. In health professions education (HPE), debates on these questions have ebbed and flowed with little sense of resolution or progress. In this article, the authors explore whether there is a problem with evidence in HPE using thought experiments anchored in Argyris' learning loops framework.From a single-loop perspective ("How are we doing?"), there may be many problems with evidence in HPE, but little is known about how research evidence is being used in practice and policy. A double-loop perspective ("Could we do better?") suggests expectations of knowledge producers and knowledge consumers might be too high, which suggests more systemwide approaches to evidence-informed practice in HPE are needed. A triple-loop perspective ("Are we asking the right questions?") highlights misalignments between the dynamics of research and decision-making, such that scholarly inquiry may be better approached as a way of advancing broader conversations, rather than contributing to specific decision-making processes.The authors ask knowledge producers and consumers to be more attentive to the translation from knowledge to evidence. They also argue for more systematic tracking and audit of how research knowledge is used as evidence. Given that research does not always have to serve practical purposes or address the problems of a particular program or institution, the relationship between knowledge and evidence should be understood in terms of changing conversations, as well as influencing decisions.

The influence of viewing time on visual diagnostic accuracy: Less is more.

BACKGROUND: Understanding the factors that contribute to diagnostic errors is critical if we are to correct or prevent them. Some scholars influenced by the default interventionist dual-process theory of cognition (dual-process theory) emphasise a narrow focus on individual clinician's faulty reasoning as a significant contributor. In this paper, we examine the validity of claims that dual process theory is a key to error reduction. METHODS: We examined the relationship between a clinical experience (staff and resident physicians) and viewing time on accuracy for categorising chest X-rays (CXRs) and electrocardiograms (ECGs). In two studies, participants categorised images as normal or abnormal, presented at viewing times of 175, 250, 500 and 1000 ms, to encourage System 1 processing. Study 2 extended viewing times to 1, 5, 10 and 20 s to allow time for System 2 processing and a diagnosis. Descriptives and repeated measures analysis of variance were used to analyse the proportion of true and false positive rates (TP and FP) as well as correct diagnoses. RESULTS: In Study 1, physicians were able to detect abnormal CXRs (0.78) and ECGs (0.67) with relatively high accuracy. The effect of experience was found for ECGs only, as staff physicians (0.71, 95% CI = 0.66-0.75) had higher ECG TP than resident physicians (0.63, 95% CI = 0.58-0.68) in Study 1, and staff had lower ECG FP (0.10, 95% CI = 0.03-0.18) than resident physicians (0.27, 95% CI = 0.20-0.33) in Study 2. In other comparisons, experience was equivocal for ECG FPs and CXR TPs and FPs. In Study 2, overall diagnostic accuracy was similar for both ECGs and CXRs, (0.74). There were small interactions between experience and time for TP in ECGs and FP in CXRs, which are discussed further in the discussion and offer insights into the relationship between processing and experience. CONCLUSION: Overall, our findings raise concerns about the practical application of models that link processing type to diagnostic error, or to specific diagnostic error reduction strategies.

Competence By Design: a transformational national model of time-variable competency-based postgraduate medical education.

Postgraduate medical education is an essential societal enterprise that prepares highly skilled physicians for the health workforce. In recent years, PGME systems have been criticized worldwide for problems with variable graduate abilities, concerns about patient safety, and issues with teaching and assessment methods. In response, competency based medical education approaches, with an emphasis on graduate outcomes, have been proposed as the direction for 21st century health profession education. However, there are few published models of large-scale implementation of these approaches. We describe the rationale and design for a national, time-variable competency-based multi-specialty system for postgraduate medical education called Competence by Design. Fourteen innovations were bundled to create this new system, using the Van Melle Core Components of competency based medical education as the basis for the transformation. The successful execution of this transformational training system shows competency based medical education can be implemented at scale. The lessons learned in the early implementation of Competence by Design can inform competency based medical education innovation efforts across professions worldwide.

Crowdsourcing a diagnosis? Exploring the accuracy of the size and type of group diagnosis: an experimental study.

BACKGROUND: The consultation process, where a clinician seeks an opinion from another clinician, is foundational in medicine. However, the effectiveness of group diagnosis has not been studied. OBJECTIVE: To compare individual diagnosis to group diagnosis on two dimensions: group size (n=3 or 6) and group process (interactive or artificial groups). METHODOLOGY: Thirty-six internal or emergency medicine residents participated in the study. Initially, each resident worked through four written cases on their own, providing a primary diagnosis and a differential diagnosis. Next, participants formed into groups of three. Using a videoconferencing platform, they worked through four additional cases, collectively providing a single primary diagnosis and differential diagnosis. The process was repeated using a group of six with four new cases. Cases were all counterbalanced. Retrospectively, nominal (ie, artificial) groups were formed by aggregating individual participant data into subgroups of three and six and analytically computing scores. Presence of the correct diagnosis as primary diagnosis or included in the differential diagnosis, as well as the number of diagnoses mentioned, was calculated for all conditions. Means were compared using analysis of variance. RESULTS: For both authentic and nominal groups, the diagnostic accuracy of group diagnosis was superior to individual for both the primary diagnosis and differential diagnosis. However, there was no improvement in diagnostic accuracy when comparing a group of three to a group of six. Interactive and nominal groups were equivalent; however, this may be an artefact of the method used to combine data. CONCLUSIONS: Group diagnosis improves diagnostic accuracy. However, a larger group is not necessarily superior to a smaller group. In this study, interactive group discussion does not result in improved diagnostic accuracy.

Demonstrating causality, bestowing honours, and contributing to the arms race: Threats to the sustainability of HPE research.

As the field of health professions education (HPE) continues to evolve, it is necessary to occasionally pause and reflect on the potential effects and outcomes of our research practices. While future-casting does not guarantee that impending negative consequences will be evaded, the exercise can help us avoid pitfalls. In this paper, we reflect on two terms that have taken hold as powerful idols in HPE research that stand above questioning and apart from critique: patient outcomes and productivity. We argue that these terms, and the ways of thinking they uphold, threaten the sustainability of HPE research-one at the level of the community and one at the level of the scholar. First, we suggest that HPE research's history of endorsing a linear and causal association ethos has driven its quest to connect education to patient outcomes. To ensure the sustainability of HPE scholarship, we must deconstruct and disempower patient outcomes as one of HPE's god-terms, as the pinnacle goal of educational activities. To be sustained, HPE research needs to value all of its contributions equally. A second god-term is productivity; it impairs the sustainability of the careers of individual researchers. Problems of honorary authorship, research output expectations, and comparisons with other fields have constructed a space where only scholars with sufficient privilege can prevail. If productivity persists as a god-term, the field of HPE research could decay into a space where new scholars are silenced-not because they fail to make important contributions, but because access is restricted by existing research metrics. These are two of many god-terms threatening the sustainability of HPE research. By highlighting patient outcomes and productivity and by acknowledging our own participation in propagating them, we hope to encourage others to recognize how our collective choices threaten the sustainability of our field.

Blueprints for Connection: A Meta-Organizational Framework for Layering Theory, Philosophy, and Praxis Within Continuing Education in the Health Professions.

ABSTRACT: As a field, Continuing Professional Development (CPD) lies at the intersection of many disciplines. Tensions can occur as scholars from fields ranging from education to quality improvement seek to advance the practices and workplaces of health care professionals. Owing to the diversity of people working to affect change within the field of CPD, it remains a very challenging space to collaborate and understand the various philosophies, epistemologies, and practice of all those within the field.In this article, the authors have proposed a meta-organizational framework for how we might re-examine theory, application, and practice within the field of CPD. It is their belief that this proposal might inspire others to reflect on how we can cultivate and invite diverse scientists and scholars using a range of theories to add to the fabric of the field of CPD.

Are we talking about practice? A randomized study comparing simulation-based deliberate practice and mastery learning to self-guided practice.

OBJECTIVES: Simulation-based technical skills training is now ubiquitous in medicine, particularly for high acuity, low occurrence (HALO) procedures. Mastery learning and deliberate practice (ML + DP) are potentially valuable educational methods, however, they are resource intensive. We sought to compare the effect of deliberate practice and mastery learning versus self-guided practice on skill performance of the rare, life-saving procedure, a bougie-assisted cricothyroidotomy (BAC). METHODS: We conducted a multi-center, randomized study at five North American emergency medicine (EM) residency programs. We randomly assigned 176 EM residents to either the ML + DP or self-guided practice groups. Three blinded airway experts independently evaluated BAC skill performance by video review before (pre-test), after (post-test) and 6-12 months (retention) after the training session. The primary outcome was post-test skill performance using a global rating score (GRS). Secondary outcomes included performance time and skill performance at the retention test. RESULTS: Immediately following training, GRS scores were significantly higher as mean performance improved from pre-test, (22, 95% CI = 21-23) to post-test (27, 95% CI = 26-28), (p < 0.001) for all participants. However, there was no difference between the groups on GRS scores (p = 0.2) at the post-test or at the retention test (p = 0.2). At the retention test, participants in the ML + DP group had faster performance times (66 s, 95% CI = 57-74) compared to the self-guided group (77 s, 95% CI = 67-86), (p < 0.01). CONCLUSIONS: There was no significant difference in skill performance between groups. Residents who received deliberate practice and mastery learning demonstrated an improvement in skill performance time.

ABSTRAIT: OBJECTIFS: La formation aux compétences techniques fondée sur la simulation est maintenant omniprésente en médecine, en particulier pour les procédures de grande acuité et de faible occurrence (HALO). L'apprentissage de la maîtrise et la pratique délibérée (ML+DP) sont des méthodes éducatives potentiellement précieuses, mais elles exigent beaucoup de ressources. Nous avons cherché à comparer l'effet de la pratique délibérée et de l'apprentissage de la maîtrise par rapport à la pratique autoguidée sur le rendement des compétences de la rare intervention de sauvetage, une cricothyroïdotomie à la bougie. MéTHODES: Nous avons mené une étude multicentrique randomisée dans le cadre de cinq programmes nord-américains de résidence en médecine d'urgence. Nous avons affecté au hasard 176 résidents en SE aux groupes de ML+DP ou de pratique autoguidée. Trois experts des voies respiratoires aveuglés ont évalué de façon indépendante la performance des compétences en BAC par examen vidéo avant (pré-test), après (post-test) et 6 à 12 mois (rétention) après la séance de formation. Le principal résultat a été le rendement des compétences après le test au moyen d'une cote globale (SRC). Les résultats secondaires comprenaient le temps consacré au rendement et le rendement des compétences au test de rétention. RéSULTATS: Immédiatement après la formation, les résultats des SRC étaient beaucoup plus élevés, car le rendement moyen s'est amélioré entre le prétest (22, IC à 95 % = 21 à 23) et le post-test (27, IC à 95 % = 26 à 28), (p < 0,001) pour tous les participants. Cependant, il n'y avait aucune différence entre les groupes sur les scores GRS (p = 0,2) au post-test ou au test de rétention (p = 0,2). Au test de rétention, les participants du groupe ML+DP avaient des temps de performance plus rapides (66 secondes, IC à 95 % = 57 à 74) que ceux du groupe autoguidé (77 secondes, IC à 95 % = 67 à 86) (p < 0,01). CONCLUSIONS: Il n'y avait pas de différence significative dans le rendement des compétences entre les groupes. Les résidents qui ont bénéficié d'une pratique délibérée et d'un apprentissage de la maîtrise ont démontré une amélioration du temps consacré aux compétences.

Does allowing access to electronic differential diagnosis support threaten the reliability of a licensing exam?

INTRODUCTION: Newer electronic differential diagnosis supports (EDSs) are efficient and effective at improving diagnostic skill. Although these supports are encouraged in practice, they are prohibited in medical licensing examinations. The purpose of this study is to determine how using an EDS impacts examinees' results when answering clinical diagnosis questions. METHOD: The authors recruited 100 medical students from McMaster University (Hamilton, Ontario) to answer 40 clinical diagnosis questions in a simulated examination in 2021. Of these, 50 were first-year students and 50 were final-year students. Participants from each year of study were randomised into one of two groups. During the survey, half of the students had access to Isabel (an EDS) and half did not. Differences were explored using analysis of variance (ANOVA), and reliability estimates were compared for each group. RESULTS: Test scores were higher for final-year versus first-year students (53 ± 13% versus 29 ± 10, p < 0.001) and higher with the use of EDS (44 ± 28% versus 36 ± 26%, p < 0.001). Students using the EDS took longer to complete the test (p < 0.001). Internal consistency reliability (Cronbach's alpha) increased with EDS use among final-year students but was reduced among first-year students, although the effect was not significant. A similar pattern was noted in item discrimination, which was significant. CONCLUSION: EDS use during diagnostic licensing style questions was associated with modest improvements in performance, increased discrimination in senior students and increased testing time. Given that clinicians have access to EDS in routine clinical practice, allowing EDS use for diagnostic questions would maintain ecological validity of testing while preserving important psychometric test characteristics.

LEADS+ Developmental Model: Proposing a new model based on an integrative conceptual review.

BACKGROUND: Leaders in academic health sciences centres (AHCs) must navigate multiple roles as an inherent component of their positions. Changing accountabilities, varying expectations, differing leadership capabilities required of multiple leadership roles can be exacerbated by health system disruption, such as during the COVID-19 pandemic. We need improved models that support leaders in navigating the complexity of multiple leadership roles. METHOD: This integrative conceptual review sought to examine leadership and followership constructs and how they intersect with current leadership practices in AHCs. The goal was to develop a refined model of healthcare leadership development. The authors used iterative cycles of divergent and convergent thinking to explore and synthesise various literature and existing leadership frameworks. The authors used simulated personas and stories to test the model and, finally, the approach sought feedback from knowledge users (including healthcare leaders, medical educators and leadership developers) to offer refinements. RESULTS: After five rounds of discussion and reformulation, the authors arrived at a refined model: the LEADS+ Developmental Model. The model describes four nested stages, organising progressive capabilities, as an individual toggles between followership and leadership. During the consultation stage, feedback from 29 out of 65 recruited knowledge users (44.6% response rate) was acquired. More than a quarter of respondents served as a senior leader in a healthcare network or national society (27.5%, n = 8). Consulted knowledge users were invited to indicate their endorsement for the refined model using a 10-point scale (10 = highest level of endorsement). There was a high level of endorsement: 7.93 (SD 1.7) out of 10. CONCLUSION: The LEADS+ Developmental Model may help foster development of academic health centre leaders. In addition to clarifying the synergistic dynamic between leadership and followership, this model describes the paradigms adopted by leaders within health systems throughout their development journey.

Education Theory Made Practical: Creating open educational resources via an apprenticeship model.

Introduction: Clinical faculty may have limited knowledge of education theories and best practices in health professions education. Many faculty development programs focus on passive learning with limited application to practice. There is a need for more active engagement for early career educators. Methods: We created an apprenticeship-based electronic book series focused on translating education theories into practical applications for clinician educators. Chapters were authored by teams of two to four geographically separated early career educators, who were tasked with explaining an education theory and relating it to their educational practice. The chapters underwent internal peer review, followed by open peer review as a blog post and eventual publication. Usage data were collected, and surveys were sent to authors and end-users. Results: Six volumes (60 total chapters) have been created to date by 180 unique authors and 17 editors over a 6-year period. There have been 65,571 total blog page views and 17,180 total book downloads across the five published volumes. Authors reported an increase in their perceived knowledge (pre 2.6 ± 1.7 vs. post 7.2 ± 1.1, mean difference 4.5/9.0, 95% confidence interval [CI] 4.0-5.0, p < 0.001) after writing their chapter. Authors also reported career benefits including authorship for academic advancement/promotion and developing an area of education theory expertise. End-users also reported a mean increase in their perceived knowledge (pre 4.4 ± 2.5 vs. post 7.3 ± 1.4, mean difference 2.9/9.0, 95% CI 2.1-3.8, p < 0.001) after reading a chapter. Conclusion: The Education Theory Made Practical electronic book series represents a proof of concept for an apprenticeship-based model to teach education theory, while also creating scholarship and open access resources for the broader community.

Implementation of Entrustable Professional Activities assessments in a Canadian obstetrics and gynecology residency program: a mixed methods study.

Background: Since the implementation of competency-based medical education (CBME) across residency training programs in Canada, there has been limited research understanding how entrustable professional activity (EPA) assessments are used by faculty supervisors and residents. Objective: This study examines how EPA assessments are used in an Obstetrics and Gynecology residency program and the impact of implementation on both groups. Methods: A mixed methods study design was used. Part one involved the aggregation of descriptive data of EPA assessment completion for postgraduate year 1 and 2 residents from July 2019 to May 2020. Part two involved a thematic analysis of semi-structured interviews of residents and faculty. Results: There was significant uptake of EPA assessments across community and teaching hospitals with widespread contribution of assessment data from faculty. However, both residents and faculty reported that the intended design of EPA assessments as low-stakes assessments to provide formative feedback is not how EPA assessments are experienced. Residents and faculty noted the increased level of administrative burden and related perceived stress amongst the resident group. Conclusions: The implementation of EPA assessments is feasible across a variety of sites. However, previous measurement challenges remain. Neither residents nor faculty perceive the value of EPAs to improve feedback, despite their intended nature.

Contexte: Depuis l'implantation de l'éducation médicale fondée sur les compétences (EMFC) dans les programmes de résidence au Canada, peu de recherches ont été menées pour comprendre comment les évaluations des activités professionnelles confiables (APC) sont utilisées par les superviseurs et les résidents. Objectif: Cette étude examine l'utilisation des évaluations des APC dans un programme de résidence en obstétrique et gynécologie et les impacts de son implantation sur ces deux groupes. Méthodes: La première partie de cette étude à méthode mixte a consisté en l'agrégation des données descriptives sur les évaluations d'APC pour les résidents de première et deuxième année effectuées entre juillet 2019 et mai 2020. Dans un second temps, nous avons analysé thématiquement les entrevues semi-structurées de résidents et de membres du corps professoral. Résultats: Il y a eu une importante participation aux évaluations des APC dans les hôpitaux communautaires et universitaires, avec une contribution considérable de données d'évaluation de la part des professeurs. Toutefois, alors que les évaluations des APC sont considérées à faibles enjeux etvisent à fournir une rétroaction formative, ce n'est pas de cette façon qu'elles sont vécues Les résidents et le corps professoral ont tous les deux rapporté une augmentation de la charge administrative et du niveau de stress perçu chez les résidents. Conclusion: L'implantation des APC est réaliste dans plusieurs sites. Malgré l'intention qui a motivé la mise en place des APC, ni les résidents ni le corps professoral ne perçoivent leur valeur comme moyen d'améliorer la rétroaction.

Electronic Diagnostic Support in Emergency Physician Triage: Qualitative Study With Thematic Analysis of Interviews.

BACKGROUND: Not thinking of a diagnosis is a leading cause of diagnostic error in the emergency department, resulting in delayed treatment, morbidity, and excess mortality. Electronic differential diagnostic support (EDS) results in small but significant reductions in diagnostic error. However, the uptake of EDS by clinicians is limited. OBJECTIVE: We sought to understand physician perceptions and barriers to the uptake of EDS within the emergency department triage process. METHODS: We conducted a qualitative study using a research associate to rapidly prototype an embedded EDS into the emergency department triage process. Physicians involved in the triage assessment of a busy emergency department were provided the output of an EDS based on the triage complaint by an embedded researcher to simulate an automated system that would draw from the electronic medical record. Physicians were interviewed immediately after their experience. Verbatim transcripts were analyzed by a team using open and axial coding, informed by direct content analysis. RESULTS: In all, 4 themes emerged from 14 interviews: (1) the quality of the EDS was inferred from the scope and prioritization of the diagnoses present in the EDS differential; (2) the trust of the EDS was linked to varied beliefs around the diagnostic process and potential for bias; (3) clinicians foresaw more benefit to EDS use for colleagues and trainees rather than themselves; and (4) clinicians felt strongly that EDS output should not be included in the patient record. CONCLUSIONS: The adoption of an EDS into an emergency department triage process will require a system that provides diagnostic suggestions appropriate for the scope and context of the emergency department triage process, transparency of system design, and affordances for clinician beliefs about the diagnostic process and addresses clinician concern around including EDS output in the patient record.

Examining the validity argument for the Ottawa Surgical Competency Operating Room Evaluation (OSCORE): a systematic review and narrative synthesis.

The Ottawa Surgical Competency Operating Room Evaluation (OSCORE) is an assessment tool that has gained prominence in postgraduate competency-based training programs. We undertook a systematic review and narrative synthesis to articulate the underlying validity argument in support of this tool. Although originally developed to assess readiness for independent performance of a procedure, contemporary implementation includes using the OSCORE for entrustment supervision decisions. We used systematic review methodology to search, identify, appraise and abstract relevant articles from 2005 to September 2020, across MEDLINE, EMBASE and Google Scholar databases. Nineteen original, English-language, quantitative or qualitative articles addressing the use of the OSCORE for health professionals' assessment were included. We organized and synthesized the validity evidence according to Kane's framework, articulating the validity argument and identifying evidence gaps. We demonstrate a reasonable validity argument for the OSCORE in surgical specialties, based on assessing surgical competence as readiness for independent performance for a given procedure, which relates to ad hoc, retrospective, entrustment supervision decisions. The scoring, generalization and extrapolation inferences are well-supported. However, there is a notable lack of implications evidence focused on the impact of the OSCORE on summative decision-making within surgical training programs. In non-surgical specialties, the interpretation/use argument for the OSCORE has not been clearly articulated. The OSCORE has been reduced to a single-item global rating scale, and there is limited validity evidence to support its use in workplace-based assessment. Widespread adoption of the OSCORE must be informed by concurrent data collection in more diverse settings and specialties.

Intrinsic or Invisible? An Audit of CanMEDS Roles in Entrustable Professional Activities.

PURPOSE: Postgraduate medical education in Canada has quickly transformed to a competency-based model featuring new entrustable professional activities (EPAs) and associated milestones. It remains unclear, however, how these milestones are distributed between the central medical expert role and 6 intrinsic roles of the larger CanMEDS competency framework. A document review was thus conducted to measure how many EPA milestones are classified under each CanMEDS role, focusing on the overall balance between representation of intrinsic roles and that of medical expert. METHOD: Data were extracted from the EPA guides of 40 Canadian specialties in 2021 to measure the percentage of milestones formally linked to each role. Subsequent analyses explored for differences when milestones were separated by stage of postgraduate training, weighted by an EPA's minimum number of observations, or sorted by surgical and medical specialties. RESULTS: Approximately half of all EPA milestones (mean = 48.6%; 95% confidence interval [CI] = 45.9, 51.3) were classified under intrinsic roles overall. However, representation of the health advocate role was consistently low (mean = 2.95%; 95% CI = 2.49, 3.41), and some intrinsic roles-mainly leader, scholar, and professional-were more heavily concentrated in the final stage of postgraduate training. These findings held true under all conditions examined. CONCLUSIONS: The observed distribution of roles in EPA milestones fits with high-level descriptions of CanMEDS in that intrinsic roles are viewed as inextricably linked to medical expertise, implying both are equally important to cultivate through curricula. Yet a fine-grained analysis suggests that a low prevalence or late emphasis of some intrinsic roles may hinder how they are taught or assessed. Future work must explore whether the quantity or timing of milestones shapes the perceived value of each role, and other factors determining the optimal distribution of roles throughout training.

Harnessing Natural Language Processing to Support Decisions Around Workplace-Based Assessment: Machine Learning Study of Competency-Based Medical Education.

BACKGROUND: Residents receive a numeric performance rating (eg, 1-7 scoring scale) along with a narrative (ie, qualitative) feedback based on their performance in each workplace-based assessment (WBA). Aggregated qualitative data from WBA can be overwhelming to process and fairly adjudicate as part of a global decision about learner competence. Current approaches with qualitative data require a human rater to maintain attention and appropriately weigh various data inputs within the constraints of working memory before rendering a global judgment of performance. OBJECTIVE: This study explores natural language processing (NLP) and machine learning (ML) applications for identifying trainees at risk using a large WBA narrative comment data set associated with numerical ratings. METHODS: NLP was performed retrospectively on a complete data set of narrative comments (ie, text-based feedback to residents based on their performance on a task) derived from WBAs completed by faculty members from multiple hospitals associated with a single, large, residency program at McMaster University, Canada. Narrative comments were vectorized to quantitative ratings using the bag-of-n-grams technique with 3 input types: unigram, bigrams, and trigrams. Supervised ML models using linear regression were trained with the quantitative ratings, performed binary classification, and output a prediction of whether a resident fell into the category of at risk or not at risk. Sensitivity, specificity, and accuracy metrics are reported. RESULTS: The database comprised 7199 unique direct observation assessments, containing both narrative comments and a rating between 3 and 7 in imbalanced distribution (scores 3-5: 726 ratings; and scores 6-7: 4871 ratings). A total of 141 unique raters from 5 different hospitals and 45 unique residents participated over the course of 5 academic years. When comparing the 3 different input types for diagnosing if a trainee would be rated low (ie, 1-5) or high (ie, 6 or 7), our accuracy for trigrams was 87%, bigrams 86%, and unigrams 82%. We also found that all 3 input types had better prediction accuracy when using a bimodal cut (eg, lower or higher) compared with predicting performance along the full 7-point rating scale (50%-52%). CONCLUSIONS: The ML models can accurately identify underperforming residents via narrative comments provided for WBAs. The words generated in WBAs can be a worthy data set to augment human decisions for educators tasked with processing large volumes of narrative assessments.

Diagnostic reasoning in cardiovascular medicine.

Research in cognitive psychology shows that expert clinicians make a medical diagnosis through a two step process of hypothesis generation and hypothesis testing. Experts generate a list of possible diagnoses quickly and intuitively, drawing on previous experience. Experts remember specific examples of various disease categories as exemplars, which enables rapid access to diagnostic possibilities and gives them an intuitive sense of the base rates of various diagnoses. After generating diagnostic hypotheses, clinicians then test the hypotheses and subjectively estimate the probability of each diagnostic possibility by using a heuristic called anchoring and adjusting. Although both novices and experts use this two step diagnostic process, experts distinguish themselves as better diagnosticians through their ability to mobilize experiential knowledge in a manner that is content specific. Experience is clearly the best teacher, but some educational strategies have been shown to modestly improve diagnostic accuracy. Increased knowledge about the cognitive psychology of the diagnostic process and the pitfalls inherent in the process may inform clinical teachers and help learners and clinicians to improve the accuracy of diagnostic reasoning. This article reviews the literature on the cognitive psychology of diagnostic reasoning in the context of cardiovascular disease.

The McMaster Advanced Communication Competencies Model for Psychiatry (MACC Model).

OBJECTIVE: Communication is a core competency for all physicians in training. In Canada, the importance of communication during residency is recognized through the CanMEDS framework. Although literature exists around teaching communication skills to residents, research in psychiatry residents is lacking. The purpose of this study was to explore how faculty members conceptualize the development of communication skills in psychiatry residents and develop a model reflecting this. METHODS: The authors used a constructivist grounded theory approach. Purposive sampling was used to select 14 faculty educators who regularly supervise psychiatry residents in a single university-based residency training program. Semi-structured interviews were conducted to explore educators' perceptions of how communication skills develop during residency. Constant comparative analysis occurred concurrently with data collection until thematic theoretical sufficiency was reached and relationships between themes determined. RESULTS: Five themes underlie the McMaster Advanced Communication Competencies model, describing the progressive development of communication abilities in residents. Three themes identify foundational sets of abilities including: refining common foundational relational abilities, developing foundational specific psychiatric communication abilities, and learning to reflect upon and manage internal reactions. These foundational pillars then allow residents to develop a personalized art of flexible psychiatric interviewing, and skillfully partner with patients in co-creating care plans. CONCLUSIONS: This research describes a preliminary communication competency model for teaching and assessing psychiatry residents. It defines the core communication abilities required for residents to progress to independent practice. Future research could explore and test the model's validity and transferability.